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PATENT 

Attorney Docket No. 401220 
IN THE UNITED STATES PATENT AND TRADEMARK OFFICE 

In re Application of: 
JEONG et al. 

Application No.: Unassigned Art Unit: Unassigned 

Filed: May 29, 2001 Examiner: Unassigned 

For: SYSTEM FOR 

MATCHING STEREO 
IMAGE IN REAL 
TIME 

SPECIFICATION, CLAIMS AND 
ABSTRACT AS PRELIMINARILY AMENDED 

Amendments to the paragraph beginning at page 1, line 25: 

In an article [Stuart Geman and Donald Geman. Stochastic relaxation, Gibbs 
distributions, and the Bayesian restoration of images. IEEE Transactions on Pattern 
Analysis and Machine Intelligence, PAMI-6(6): 72 1-741, Nocmbc r November 1984], a 
stereo matching method using Markov random fields and stochastic optimization 
methods, based on simulated annealing presented by S. Kirkpatrick et al., "Optimization 
by Simulated Annealing", Science, May 1983, pg. 671-680, is described. This has been 
further developed by others, for example, Geiger and Girosi using mean field theory. 
However, this class of methods is iterative in nature resulting in very long computational 
times that are not suitable for real time stereo matching. 

Amendments to the paragraph beginning at page 2, line 3: 

In an article [H. H. Baker and T. O. Binford. Depth from edge and intensity based 
stereo. In Proceedings of the International Joint Conference on Articifical Artificial 
Intelligence, page 631-636, Vancouver, Canada, 1981] and an article [Y. Ohta and T. 
Kanade. Stereo by intra- and inter-scan line search. IEEE Transactions on Pattern 
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Analysis and Machine Intelligence, PAMI-7(2):139-154, March 1985], stereo matching 
methods based on dynamic programming (DP) and heuristic post-processing are 
described. In an article [Ingemar J. Cox, Sunita L. Hingorani, Satish B. Rao, and Bruce 
M. Maggs. A maximu m lildihood likelihood stereo algorithm. Computer Vision and 
Image Understanding, 63(3):542-567, May 1996] and an article [Stan Birchfield and 
Carlo Tomasi. Depth discontinuities by pixel-to-pixel stereo. In Proceeding of the IEEE 
International Conference on Computer Vision, pages 1073- 1080m, Bombay, India, 1998], 
single-level DP in discrete pixel oriented methods are described. In an article [Peter N. 
Belhumeur. A Bayesian approach to binocular stereopsis. International Journal of 
Computer Vision, 19(3):237-260, 1996], a more complex DP method with sub-pixel 
resolution is described. Though this class of methods is much faster than the Markov 
random field based ones, they do not scale well for parallel processing and are thus still 
unsuitable for real-time stereo matching. 

Amendments to the paragraph beginning at page 5, line 28: 

An image input to the left and right cameras 10 and 1 1 is converted into digital 
signals in the form of pixels in the image processing unit 12 and one scan line of each 
image is provided to the SMC 13 in units of a pixel. After the scan line is fully provided 
to the SMC 13, disparity data is output in units of a pixel. The process in which a 
disparity is output is repeated for all scan lines of the pair of images in the same way. 
Therefore, only the process for processing a pair ofra scan4if* e lines will now be 
explained. 

Amendments to the paragraph beginning at page 6, line 3: 

As shown in FIG. 2, the SMC 13 contains a linear array of N identical processing 
elements 22 and two linear arrays, each of N/2 image registers 20 and 21 . Here, N is a 
multiple of 2. 
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Amendments to the paragraph beginning at page 6, line 25: 

Therefore, half of the processing elements 22 and half of the image registers (20 or 
21) operate at every each system clock cycle, beginning from the even-numbered 
processing elements 22 and right image registers 20. The processing step is controlled by 
read/write signal (F/B or R/W, hereinafter referred to as "RAY"). When an R/W signal 
line is in a high state, data is written and when the R/W signal line is in a low state, data is 
read. 

Amendments to the paragraph beginning at page 7, line 6: 

In the last ClkO in the initializing process, after the first half of the data in the scan 
line of the right image is input to the processing elements 22, the first pixel in the scan 
line of the left image is input to the processing elements 22. At this time, registers inside 
each processing element 224 s are set to an appropriate initial value. The initial value of 
the processing element 0 is '0 1 and the initial value of all the other processors is the 
maximum (or close to the maximum) possible value. Then, the processing process is 
continuously applied to all pixel data input at each system clock-^li H until data in the 
present scan line is all processed (ClkE is for the left image, and ClkO is for the right 
image). 

Amendments to the existing claims: 

1 . (Amended) A real-time stereo image matching system comprising: 
a=signal converting means for converting an image input from a first camera and a 
second camera into a digital signal; and 

a**=image matching means for calculating a determined matching cost based on a 
pair of pixels in one scan line of the first and second digital image signals, tracing^© 
decided a decision value which determines=4he_a minimum matching cost, and outputting 
the decided decision value as an estimated disparity according to predetermined 
activation information. 
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2. (Amended) The real-time stereo image matching system of claim 1, wherein 
the imago input to the oignal converting moano is obtained by the first camera and second 
camera4hat have optical axes parallel to each other and have4h e coplanar focal planes on 



3. (Amended) The real-time stereo image matching system of claim 1, wherein 
in the image matching means, the calculation of the matching cost is calculated after 
occlusion information^ for pixels whic h pixclo do not match in the scan line is added to 
th e firot pixel and the oocond pixel pair of pixels . 

4. (Amended) The real-time stereo image matching system of claim 1, wherein 
the image matching means further comprises: 

a=first storage means for storing the digital image pixels from the first camera; 
^second storage means for storing the digital image pixels from the second 
camera; 

a=processing means for outputting a predetermined estimated disparity using pixels 
input from the first otorago mcano and second storage means; and 

SFclock control means for providing a clock signal for controlling=the operations of 
the first and second storage means and the processing means. 

5. (Amended) The real-time stereo image matching system of claim 4, wherein 
the syste m io formed o f includes N processing means, N/2 first storage means, and N/2 
second storage means (Here, where N is=a an integer multiple of 2^. 

7. (Amended) The real-time stereo image matching system of claim 5, wherein 
among the N processing means, only=tfee_a processing means that outputs= a the 
predetermined disparity is activated at one time and the remaining processing means are 
in high impedance states. 
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, 10. (Amended) The real-time stereo image matching system of claim 4, wherein 
the clock control means outputs a first clock signal for4h§ even-numbered processors and 
the second storage means, and a second clock signal for=the odd-numbered processors and 
the first storage means. 

1 1 . (Amended) The real-time stereo image matching system of claim 4, wherein 
the processing means comprises: 

a forward processor for receiving a pixel of one scan line in the first storage means 
and the second storage means, and outputting a determined matching cost and a decision 
value; 

^decision storage means for storing the decision value output from the forward 
processor; and 

a second processor for outputting a determined disparity, using the decision value 
output from the decision storage meanSi according to the determined activation 
information. 

12. (Amended) The real-time stereo image matching system of claim 11, 
wherein^ when a write control signal is input from outside, the first processor operates, 
and when a read control signal is input from outside, the second processor operates. 

13. (Amended) The real-time stereo image matching system of claim 11, wherein 
the decision storage means has a last-in first-out structure in which the dooidod decision 
value that is output last from the first processor is first input to the second processor. 

14. (Amended) The real-time stereo image matching system of claim 11, wherein 
the first processor comprises: 

a=matching cost calculating means for calculating a matching cost, using a pixel of 
one line in the first storage means and the second storage means; 

a=first adding means for adding the calculated matching cost to the fed-back 
accumulated cost; 
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a=comparing means for comparing the output of the first adding means with the 
costs of neighboring processing means, and then outputting the minimu m matches of cost 
and decision value; 

^storage means for storing the minimum cost that is the rcoult o f a comparison 
produced by the comparison means , as the accumulated cost; and 

^second adding means for adding the entire cost and occlusion cos t to produce a 
sum , and then outputting th e rcoult sum to neighboring processing means. 

15. (Amended) The real-time stereo image matching system of claim 11, wherein 
the second processor comprises: 

a=logical OR means fo r performing OR-ing=A§ activation information of the 
neighboring processing means and=the feed-back activation information route; 

a register for storing the last activation information and that io the rcoult of 
produced by the OR. ing logical OR means ; 

demultiplexing means for demultiplexing the last activation information 
according to the decision value output from the decision storage means to outpu t and 
outputting to the neighboring processing means and=feed feeding back to the logical OR 
means; and 

a tri-state buffer for outputting the decision value output from the decision storage 
means, as a determined disparity, according to the activation information of the register. 

Amendments to the abstract: 

Abstract 

A system for processing stereo matching of a video image sequence in a real-time 
mode io provided . The system includes a signal converting mcano converter for 
converting an image input from a first camera and a second camera into a digital signal; 
and an image matching mcano chip for calculating a determined matching cost based on a 
pair of pixels in one scan line of the first and second digital image signals, tracing the 
decision value which determines the minimum matching cost, and outputting the decided 
value as an estimated disparity according to determined activation information; and a 
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display mcano for displaying the output from the image matching mcano . According to 
the system, real-time stereo matching is enabled? by parallel processing or video image 
sequences using an algorithm which is based on a new dynamic trellis based method and 
is optimal in the Bayesian sense. 
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